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DATA  ANALYSIS  OF  THE 

TEACHING  AWARD  BALLOTS 

by 

Robert  R.  Read 

I.   INTRODUCTION 

The  annual  award  for  excellence  in  teaching  was  established 
by  the  Superintendent  in  1969.   The  first  recipient  was  honored 
at  the  Spring  graduation  in  June  19  70.   There  is  a  cash  prize 
(which  has  become  substantial  due  to  an  endowment  by  Rear  Admiral 
John  J.  Schieffelin)  and,  in  recent  years,  the  committee  has 
provided  the  Provost  with  a  list  of  honorable  mention  faculty 
who,  as  a  result,  have  received  step  increments  in  pay.   Further, 
the  winning  instructor's  name  is  etched  on  a  plaque  in  the  library, 

The  recipient  of  the  award  is  chosen  by  a  committee 
of  faculty  which  conducts  a  secret  poll  of  on  board  students 
and  recent  former  students.   The  data  produced  by  the  poll 
are  machine  processed.   The  output  is  coded  so  that  the  numer- 
ical summaries  cannot  be  associated  with  the  names  of  the 
particular  faculty  involved,  and  the  winner  is  selected  in  a 
totally  objective  fashion.   The  membership  of  the  committee 
rotates.   Each  new  committee  has  exercised  its  responsibilities 
in  terms  of  defining  the  set  of  faculty  eligible  to  receive  the 
award  and  the  set  of  franchised  voters  to  participate  in  the 
poll.   Also,  the  specific  use  made  of  the  numerical  summaries 
may  change  from  committee  to  committee.   Major  changes  in  the 
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structure  of  the  ballot  have  been  avoided  since  such  changes 
would  entail  substantial  changes  in  the  computer  codes. 

The  basic  format  of  the  ballot  was  set  by  the  initial 
committee  as  was  the  nature  of  the  data  summaries  and  the  author 
played  a  substantial  role  in  these  activities.   Over  the  years, 
the  uses  made  of  the  balloting  data  have  increased  (i.e.  an 
honorable  mention  list  is  extracted  and  previous  year's  per- 
formance data  is  used  in  selecting  the  winner)  and  the  time 
came  for  revising  the  data  processing  effort  in  the  light  of 
this.   Because  of  this  need  the  author  was  placed  again  on 
the  selection  committee  (1978)  and,  after  study,  recommended 
revisions  in  processing,  monitoring,  and  record  keeping.   The 
purpose  of  the  present  report  is  to  document  the  support  for 
and  nature  of  the  changes. 

The  basic  recommendations  are  fourfold: 
i)   Modify  the  basic  scoring  system  so  that 

(a)  the  correlation  between  high  scoring  instructors 
and  instructors  who  are  identified  on  few  ballots 
is  removed; 

(b)  the  practice  of  listing  large  numbers  of  comparable 
faculty  on  a  ballot  in  an  effort  to  boost  the 
prime  candidate  is  discouraged. 

11)   Prepare  paired  comparison  studies  in  parallel  to  the 

basic  scoring  system  so  that  attention  is  drawn  to  those 
instructors  whose  basic  scores  are  relatively  low  because 
the  set  of  instructors  with  whom  they  were  compared  was 
an  unusually  strong  set. 
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iii)   Organize  the  record  keeping  process  so  that  there  is 
a  one-to-one  correspondence  between  instructors  and 
their  identification  numbers.   Maintain  historical 
records  of  each  instructor's  performance  in  previous 
years'  balloting  coded  by  A,  B,  E,  I,  representing, 
respectively,  top  5%,  next  15%,  eligible  but  not  in 
top  20%,  ineligible, 
iv)   Monitor  the  data  each  year  so  that  the  goals  in  i)  and 
ii)  are  maintained.   Perform  ballot  population  studies 
so  that  the  selection  committee  will  be  made  aware  of 
anomalies  in  voting  patterns. 
The  organization  of  the  report  is  as  follows.   Section  II 
contains  some  historical  information  about  the  structure  of 
the  ballot,  control  of  errors,  and  choice  of  notation.   In 
Section  III  is  described  the  early  approach  to  summarizing 
the  data,  scoring,  and  choosing  the  winner.   The  pertinent 
experiences  and  biases  observed  in  the  data  processing  are 
described  in  Section  IV.   A  paired  comparison  study  of  the 
data  is  given  in  Section  V.   This  leads  to  Section  VI  which 
summarizes  the  results  and  describes  the  newly  implemented 
system.   Four  appendices  are  included  which  contain  many  of 
the  supporting  details  and  data  analyses. 


II.   BACKGROUND  AND  NOTATION 

The  first  Selection  Committee  (chaired  by  Prof.  T.  Gawain) 
struggled  with  the  issues  of  what  data  to  collect  and  how  to 
collect  it.   It  decided  to  gather  information,  by  secret  ballot, 
largely  from  the  consumers  of  instruction.   Originally,  the  set 
of  franchised  voters  consisted  of  students  on  board,  faculty, 
curricula  officers,  and  recent  alumni.   Each  voter  was  asked  to 
specify  the  population  of  instructors  (from  a  list  of  eligible 
instructors)  that  they  felt  competent  to  compare,  and  from 

this  population  they  were  asked  to  name  their  first,  second, 
and  third  choices.   The  ballot  had  to  list  at  least  five 
instructors  in  order  to  qualify  for  inclusion  in  the  system. 
Also  some  biographies  were  collected  such  as  voter  category 
(student,  alumni,  faculty,  curricular  officer)  and  curricular 
area  (aeronautical  engineering,  operations  research,  etc.). 
Each  ballot  contains  space  for  voluntary  written  remarks 
supporting  its  first  choice. 

The  ballots  are  reviewed  for  general  validity  by  a 
neutral  source  and  the  data  are  transferred  to  IBM  punched 
cards.   Each  ballot  is  given  an  index  number  so  that  it  may 
be  retrieved  if  necessary.   The  written  comments  are  coded 
for  the  computer  only  in  terms  of  their  presence  or  absence. 
The  eligible  instructors  are  coded  with  a  four  digit  ID 
number.   The  first  three  digits  are  an  index  and  the  last  is 
a  parity  check  used  to  catch  keypunch  errors.   The  parity 


check  involves  matching  this  digit  with  the  last  digit  of  the 
dot  product  of  the  first  three  digits  in  the  ID  with  the  vector 

(3,5,1) • 

It  is  useful  to  think  of  the  returned  ballots  as  a 
huge  data  matrix.   The  rows  correspond  to  the  eligible  faculty 
and  each  column  represents  a  ballot.   Thus  each  column  contains 
at  least  five  entries  with  a  one  opposite  the  row  matching  the 
faculty  member  receiving  the  first  place  vote,  a  two  in  the 
row  representing  the  second  choice,  a  three  in  the  row  repre- 
senting the  third  choice,  and  some  mark  in  all  positions  cor- 
responding to  other  faculty  identified  by  that  ballot.   All 
other  positions  remain  void. 

At  the  right  border  of  this  matrix  one  can  tally  four 
quantities   X,,  X~  ,  X_,  N   for  each  instructor.   For   i  =  1,2,3 

the  value   X.   is  the  number  of  votes  of  rank  i  received,  and 
1 

N   is  the  number  of  ballots  that  identify  the  particular 
instructor.   These  four  quantities  have  been  used  to  create 
a  score  of  the  form 


(w  X   +  w2X2  +  w  X  )/N  (2.1) 


where   w, ,  w~  ,  w~   are  nonnegative  weights  attached  to  the 
values  of  first,  second,  and  third  place  votes,  respectively. 
The  early  computer  programs  use  the  numbers  4,  2,  1  for  the 
weights.   Also,  for  each  instructor,  is  recorded  the  number   D 
of  ballots  that  contain  a  statement  supporting  his  nomination 
for  first  place.   Clearly   D  <_  X-,   for  each  instructor. 


At  the  bottom  border  of  this  data  matrix,  one  can  tally 
the  number   K  of  eligible  faculty  identified  on  each  ballot. 
These  numbers  were  used  for  monitoring  purposes  to  see  whether 
the  voters  in  the  various  categories  and  curricular  areas  had 
the  same  distributions  for  number  of  instructors  identified 
per  ballot.   Further  use  will  be  described  later. 

It  is  useful  to  comment  on  the  extreme  inbalance  of 
the  data.   If  all  ballots  identified  the  same  number  of  instruc- 
tors and  all  instructors  were  identified  by  the  same  number 
of  ballots  then  a  highly  defensible  selection  procedure  could 
be  devised.   In  reality  we  are  a  long  way  from  this  ideal. 
Some  arbitrary  choices  must  be  made  and  the  performance  of 
the  system  must  be  monitored. 


III.   EARLY  APPROACH  TO  BALLOT  DATA  ANALYSIS 

The  score   S   was  used  to  rank  the  instructors  in 
decreasing  order.   This  score  did  not  necessarily  determine 
the  winner.   It  was  used  primarily  as  a  device  to  order  the 
instructors  so  that  the  strongest  ones  appeared  together  at 
the  top  of  the  list. 

The  score  formula  (2.1)  carries  some  presumptions  with 
it.   Division  by   N   presumes  that  a  good  faculty  member's 
ability  to  poll  ranking  votes,  i.e.  the   X.,  is  proportional 
to   N   the  number  of  ballots  that  identify  him.   The  value   N, 
in  turn,  is  thought  to  be  roughly  proportional  to  the  number 
of  students  he  has  taught.   Thus  division  by   N   was  viewed 
as  an  equilizer  for  the  problem  of  unequal  exposure  to  students 
by  faculty.   The  system  of  weights  represents  the  worth  of 
first,  second,  and  third  place  votes  relative  to  each  other. 
The  choice  (4,2,1)  was  made  arbitrarily  and  not  without  con- 
siderable dissent.   Several  alternative  weighting  systems  were 
proposed  including  the  set   (3,2,1)   which  is  the  one  offered 
by  Condorcet,  Laplace,  and  others  in  theories  of  elections 
(see  [1,4]) .   Note  that  only  the  relative  proportions  of  the 
weights  are  important,  not  their  individual  values.   Clearly 
the  set  should  satisfy  the  constraint 

wl  2.  w9  -  w3  -  °  (3.1) 

The  set  (3.1)  will  be  called  the  set  of  admissible  weights. 


Historically,  the  first  selection  committee  agreed 
rather  quickly  that  it  was  appropriate  to  divide  each   X.^ 
by   N.   It  had  difficulty  in  deciding  what  to  do  next.   At 
this  point  the  data  summary  consisted  of  a  vector  (for  each 
instructor) 

x  =  (x  ,  x2 ,  x_)  (3.2) 

in  the  first  octant  of  three  space,  where   x.  =  X.  t  N. 
The  goal  is  to  choose  one  of  these  points  as  the  'best' 
one  in  an  agreeable  and  defensible  way.   Generally  there  are 
two  or  three  hundred  of  these  points. 

The  following  screening  procedure  was  applied  and 
successfully  reduced  the  number  of  points  under  consideration 
to  a  handful.   The  idea  (borrowed  from  game  theory)  is  to 
eliminate  from  consideration  all  eligible  faculty  who  are 
"dominated."   An  instructor's  score  vector   x'   is  said  to 
be  dominated  if  there  exists  another  score  vector   x"   such 
that  the  projected  (one-dimensional)  scores  using  Equation  (2.1) 
satisfy 

S'  <  S"   for  all  admissible  weights.  (3.3) 

HISTORICAL  NOTE;   The  original  computer  codes  performed  this 
screening  in  two  stages.   The  first  stage  searched  for  vectors 
x"   that  dominated   x'  "absolutely."   That  is 

xl  £  xi'      x2  £  x2'      x3  £  x3 


with  strict  inequality  in  at  least  one  of  the  three  positions. 
The  second  stage  considered  only  the  remaining  score 
vectors  and  eliminated  those  that  were  "completely  dominated" 
according  to  (3.3).   This  is  readily  accomplished  as  the  set 
of  admissible  weights  (normalized  so  that   w   +  w   +  w   =1) 

-L  ^  O 

is  a  convex  set  having  extreme  points   (1,0,0),  (1/2,  1/2,  0), 
and  (1/3,  1/3,  1/3)  .   Since  a  convex  set  is  determined  by 
its  extreme  points,  and  because  of  the  separating  hyperplane 
theorem,  the  inequality  (3.3)  need  only  be  checked  at  these 
three  points.   If   S'  <  S"   at  all  three  points  then   S* 
is  completely  dominated  and  removed  from  further  consider- 
ation.  The  committee  can  restrict  its  attention  to  the  un- 
dominated  instructors. 

The  data  from  the  first  year  of  the  award  are  interest- 
ing.  The  screening  by  absolute  dominance  resulted  in  thirteen 
eligible  teachers  from  an  original  list  of  249.   Further 
screening  using  complete  dominance  reduced  this  to  three. 

The  three   x   vectors  and  corresponding  values  of 
N  appear  below. 


Instructor 


T. 


N 

Xl 

X2 

X3 

58 

.483 

.172 

.0  86 

59 

.508 

.136 

.051 

106 

.509 

.123 

.047 

(0,  0,  1) 


(0,  1,0) 


Figure  3.1 
PARTITION   OF  THE    SET   OF   ADMISSIBLE  WEIGHTS 
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The  subscripts  of  the  instructors  represent  ranks  according  to 
the  initial  ordering.   If  the  weights  are  allowed  to  "float" 
then  each  of  the  three  remaining  instructors  can  achieve  the 
highest  score  depending  upon  how  the  weights  are  chosen.   The 
choices  are  presented  in  Figure  3.1  using  barycentric  coordi- 
nates ([10])  .  The  coordinate  system  is  the  simplex   (w,  ,  w?  ,  w->! 

constrained  by  w,  +  w„  +  w  =  1   and  all   w.  >  0.   The  smaller 
1       1     2     3  l  — 

right  triangle  is  the  set  of  admissible  weights  (3.1).   The 
three  line  segments  serve  to  illustrate  the  weights  that  pro- 
duce the  equality  of  the  pairs  of  scores   S-  =  S~,  S~  =  S,  , 
and   S~  =  S, .   The  two  outside  lines  partition  the  right 
triangle  into  three  subsets  which  determine  the  decision  rule. 
If  the  weight  vector  is  chosen  in  the  left  subset  then   T_ 
is  the  winner.   It  it  is  chosen  in  the  right  subset  then   T, 
is  the  winner,  and  otherwise   T2   is  the  winner.   It  is  noted 


tha 


t  line   S~  =  S,   does  not  affect  the  decision  partition 


because   T.   can  win  only  if   S.  >  S.   for  both  values  of 
l  l     j 

j  ^  i.   For  reference  purposes  a  quadrilateral  has  been  drawn 
depicting  the  region  where  the  relative  weights  satisfy  the 
inequalities 

1/3  <_  w2/w1  <_  2/3,    1/3  <_   w3/w2  <_   2/3   . 

The  weighting  scheme  having  the  ratio  4:2:1  is  at  the  center 

of  this  quadrilateral. 

By  using  this  screening  procedure  the  committee  can 
reduce  its  choices  to  a  few  (in  this  case,  three)  and  make 
its  selection  without  directly  choosing  a  set  of  weights. 
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It  can  also  seek  further  information  about  the  choices.  For 
example  the  quality  of  the  supporting  statements  can  be  (and 
has  been)  examined. 

Returning  to  the  data  of  the  first  year  of  the  award, 
a  modest  "paired  comparison"  analysis  provided  the  selection 
committee  with  some  additional  information  about  the  instruc- 
tors  T, ,  T~ ,  and  T\.   There  was  only  one  ballot  that  iden- 
tified both  T,   and  T-   and  it  made  no  preference.   There 
were  19  ballots  that  identified   T,   in  conjunction  with   T? 
and  13  other  ballots  that  identified  both   T„   and   T_.   These 
"head- to-head"  comparisons  provided  the  information  that  gen- 
erally  T~   was  preferred  to   T_.   by  the  13  ballots  and  that 
T-.   was  preferred  to   T,   by  the  19  ballots.   Some  additional 
support  for   T„   was  found  by  including  some  information 
contained  in  further  paired  comparisons  among  the  "top  twenty" 
teachers  as  defined  by  the  score  (2.1)  using  4:2:1  weights. 
An  arbitrary  rule  was  adopted  that  allowed  direct  comparisons 
only  if  at  least  five  ballots  identified  both  members  of  the 
pair.   (One  exception  was  allowed  in  the  case  of  the  three 
ballots  that  unanimously  preferred   T-.   to   T.  .  )   The  result 
was  a  set  of  "strings"  ("~"  denotes  "tied  score,  non-zero" 
and  ">"  denotes  "is  preferred  to") 


T   >  T    ~  T 
2     15     3  ' 


T2    T5  >  T3  >  T4  >  TX1  >  T1#    etc 


with  virtually  no  conflicting  preference  patterns 
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This  information  was  made  available  to  the  selection  committee 
for  use  as  they  saw  fit.   The  complete  data  for  these  paired 
comparisons  is  contained  in  Ref.  [7].   The  vote  of  the 
selection  committee  resulted  in  the  choice  of   T„.   Only  at 
this  point  was  the  winner's  name  unveiled. 

The  data  were  studied  for  other  characteristics 
(Ref.  [7,2])  and  a  few  are  mentioned  now.   A  principal  com- 
ponent analysis  of  the   x,  ,  x„  ,  x~,  revealed  that  the  direc- 
tion (in  weight  space   w)  of  maximum  variance  is  very  nearly 
4:2:1.   Using  these  weights,  the  distribution  of  positive 
scores  is  exponential,  specifically 


P(S  >  z|S  >  0}  =  e  2z 


The  distributions  of   N   (number  of  ballots  that  identify 
particular  faculty)  and  K   (number  of  faculty  identified 
on  the  ballots),  are  both  skewed  positively.   For  1970, 
the  means  and  standard  deviation  are 

y (N)  =  64.5  a(N)  =  40.4 

y (K)  =  15.5  o(K)  =  10.5 

About  50%  of  the  on-board  students  returned  their  ballots. 
Chi-square  contengency  table  analyses  showed  no  evidence  of 
dependence  of  score  on  either  voter  category,  or  on  the 
instructor's  academic  rank. 
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IV.   EXPERIENCE,  TRANSITION  AND  RECENT  STUDIES 

In  each  of  the  first  four  years  that  this  system  was 
in  use  a  master's  thesis  was  written  [2,3,7,11]  which  served 
to  monitor  the  process  and  look  for  anomalies.   A  major 
anomaly  that  was  noticed  early  was  that  many  faculty  voters 
tended  to  identify  excessively  large  numbers  of  other  faculty 
on  their  ballots.   This  was  viewed  as  disingenuous  and,  by 
1975,  resulted  in  the  disfranchisement  of  faculty  as  voters. 
At  about  the  same  time  it  was  decided  to  invalidate  ballots 
which  identified  too  many  faculty.   Specifically,  the  voters 
were  instructed  to  identify  a  number   K   of  eligible  faculty 
in  the  range 

5  <_  K  <_  25  (4.1) 

Also,  since  the  curricular  of ficers  usually  are  also  alumni, 
they  were  removed  from  the  voter  category  list. 

The  selection  committees  of  1973-74  recognized  that 
for  each  winner  there  were  a  substantial  number  of  others 
equally  deserving  of  recognition.   They  recommended  that 
multiple  awards  be  made,  but  this  was  rejected  by  the  admini- 
stration.  Instead,  the  custom  began  of  identifying  a  set  of 
ten  or  so  "honorable  mentions"  and  these  faculty  have  always 
been  recognized  with  pay  step  increases. 

As  more  data  and  experience  become  available,  the 
selection  committee  began  to  include  "past  performance" 
data  in  making  their  selection.   In  recent  years  there  has 
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been  a  very  strong  tendency  to  choose  the  winner  from  among 
those  high  scoring  faculty  who  also  have  a  good  historical 
record  in  the  honorable  mention  list.   This  practice  has  intro- 
duced some  rather  unfortunate  biases  since  the  record  keeping 
process  did  not  keep  track  of  the  corresponding  historical 
eligibility  of  faculty.   Moreover  it  was  noticed  that  the 
honorable  mention  list  had  a  bias  favoring  faculty  with  small 
values  of   N   (number  of  ballots  identifying) .   Such  a  bias 
leads  to  a  tendency  of  favoring  those  who  do  not  teach  many 
students . 

In  order  to  respond  to  these  problems  it  was  necessary 
to  perform  some  exploratory  data  analysis  to  complete  ballot 
data.   Unfortunately  no  prior   year's  data  had  been  saved.   The 
author  undertook  this  work  at  the  close  of  the  19  78  balloting 
and  those  data  as  well  as  the  current  1979  data  were  studied. 
The  important  things  learned  are  described  next. 

Although  the  selected  winner  is  not  necessarily  the 
individual  with  the  largest  of  the  aforementioned  scores,  S, 
the  top  ten  have  been  determined  by  this  score  and  the 
arbitrariness  of  this  score  has  come  under  scrutiny.   Moreover, 
as  already  stated, this  scoring  system  tends  to  favor  faculty 
having  the  smaller  values  of   N.   This  is  illustrated  in 
Figure  4.1  which  contains  scatter  plots  of  score  (using  4:2:1 
weights)  against   N.   The  top  diagram  shows  the  bias  favoring 
instructors  with  small   N.   The  bottom  diagram  shows  how  the 
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scatter  plot  would  look  if  no  normalization  by   N  were 
applied.   Thus,  under  the  current  system,  the  less  well-known 
instructors  have  the  advantage. 

Some  studies  have  been  made  in  an  effort  to  correct  this 
situation.   First,  a  modified  score  of  the  form 

S'  =  (w1X1  +  w2X2  +  w3X3)/NP       for   0  <_   p  <_   1     (4.1) 

was  considered  for  several  values  of  the  exponent   p   and 
several  sets  of  weights.   According  to  this  model  a  faculty 
member's  ability  to  accumulate  ranking  votes  is  an  increasing 
concave  function  of   N.   The  effect  is  illustrated  in  Figure  4.2 
which  contains  scatter  plots  of   S '  vs  Np   for  values  of 
p  =  .75,  .5,  .25  (and  the  same  4:2:1  weights).   Interestingly, 

it  turns  out  that  the  exponent   p  =  . 75   does  a  good  job  of 
making  the  high  scores  uncorrelated  with   N   and  this  result 
appears  to  hold  for  a  rather  large  spectrum  of  weights.   Support 
for  this  point  is  contained  in  Appendix  A. 

Deeper  considerations  uncovered  the  fact  that  the 
ballots  contain  more  information  than  was  being  used.   Specifically 
it  was  noticed  that  under  the  present  scoring  system  a  first  place 
vote  (or  any  other  rank  for  that  matter)  counts  the  same  regard- 
less of  the  number  of  faculty  with  whom  he  is  compared,  i.e.  it 
is  the  same  whether  that  individual  is  the  best  of  five  or  the 
best  of  twenty-five.   Clearly  the  latter  condition  provides  more 
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FIGURE  4.1 
Scatter  Plots  of   N  vs  Score  for   p  =  1,0 
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FIGURE    4.2 

Scatter   Plots    of      N    vs    Score    for      p   =    .75,     .5,     .25 
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information  than  the  former  and  it  is  proposed  that  this  be 
reflected  in  the  accounting  system.   Specifically  let  us  view 
each  ballot  as  a  rank  distribution.   If  a  ballot  identifies  a 
total  of   K   instructors,  then  the  top  ranked  individual  on  that 
ballot  represents  the   K/(K+l)th  quantile  of  that  distribution, 
the  second  ranked  individual  represents  the   (K-l) / (K+l) th 
quantile,  and  the  third  the   (K-2) / (K+l) th  quantile.   All  others 
must  be  viewed  as  tied  for  positions  4  through   K   and  they  are 
awarded  the  average  rank,  i.e.,  the   (K-2) /2 (K+l) th  quantile. 
Using  this  system  each  ballot  can  contribute  more  information 
to  the  data  summary  and  to  the  score  of  each  of  its  identified 
instructors . 

More  specifically,  the  new  information  collected  for 
each  instruction  consists  of   N   (as  before)  and 

Z1  =  I   K/(K+1) , 

Z2  =  I     (K-1)/(K+1) 

Z3  =  I    (K-2)/(K+l) 

Z4  =  I    (K-2)/2(K+l) 

where  the  sums  are  taken  over  all  ballots  that  identify  that 
instructor.  With  this  change,  a  new  weighted  scoring  system 
is  proposed,  viz. , 


S   =  (w1Z1  +  w2Z2  +  w3Z3  +  w4Z4)/NP  (4.3) 


(4.2) 
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* 
The  behavior  of  the  scores   S    was  studied  (using  the  19  78  data) 

for  the  weighting  systems  (3:2:1:0),  (5:4:3:1),  (4:3:2:1), 

(1:1:1:0),  and  (1:1:1:1)  and   p  =  .75,  .5  and  .25.   Figure  4.3 

*      P 
contains  the  scatter  plots  of   S   vs  1ST   for  the  5:4:3:1  weights. 

Again   p  =  . 75   produces  scores  whose  large  values  are  uncorre- 
lated  with   N.   Also  this  result  appears  to  be  robust  as  the 
weighting  system  is  allowed  to  vary  over  the  above  listed  systems 
Appendices  A  and  B  contain  further  data  supporting  these  points. 

Although  the  scatter  plots  exhibit  shape  that  does 
not  depend  on  the  weights,  the  system  of  weights  is  quite 
important  in  determining  exactly  who  becomes  an  "honorable 
mention"  and  exactly  what  is  placed  in  the  "past  performance" 
record  of  faculty.   Since  the  newly  proposed  score  has  these 
arbitrary  inputs  it  is  important  to  monitor  its  behavior  over 
time  and  adjust  the  inputs  if  necessary  so  that  the  high  scores 
remain  uncorrelated  with   N.   Further,  since  the  inputs  are 
arbitrary  so  are  the  resulting  rankings  of  faculty,  and  the  past 
performance  record  should  not  contain  either  score  or  rank  but 
a  general  index  or  grade  of  performance.   Since  the  system  is 
geared  to  identify  good  instructors  it  is  recommended  that  a 
number,  say  5%,  be  designated  as  class  A  instructors  and  the 
next  layer,  say  15%,  be  remembered  as  class  B  instructors.   The 
remaining  eligible  faculty  should  be  so  marked  and  the  values 
of   N   should  be  recorded  for  all.   The  question  of  how  to 
designate  the  class  A  and  B  instructors  is  discussed  later. 
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75,  .5,  .25 


21 


V.   PAIRED  COMPARISONS 

The  proposed  scoring  system  utilizes  more,  but  not 
all,  of  the  information  contained  in  the  ballots.   Since  each 
ballot  makes  some  direct  comparisons  between  individual  faculty- 
it  should  be  possible  to  gather  and  summarize  this  direct 
information.   The  construction  of  a  square  win-loss  matrix 
is  useful  to  this  end.   The  order  of  this  matrix  is  the  number 
of  eligible  faculty.   The  entry  in  row  i  and  column  j  is  the 
number  of  ballots  that  rate  instructor  i  higher  than  instructor  j. 
Ties  (i.e.  both  i  and  j  identified  on  a  ballot  but  neither 
awarded  a  ranking  position)  count  one-half  each.   All  diagonal 
(i  =  j)  entries  are  zero. 

The  first  step  in  analyzing  a  win-loss  matrix  is  to 

identify  the  subsets  of  comparable  instructors.   Thus  the  list 

of  instructors  must  be  partitioned  into  several  sets.   If  two 

instructors   (i,j)   appear  in  the  same  set  then  there  exists  a 

string  (possibly  empty)  of  instructor  indices   (k, ,  ...  ,  k  ) 

such  that  all  consecutive  pairs  in  the  string  i,  k, ,  k~,...,  k  ,  j 

are  directly  comparable  (i.e.,  either  the   (k  ,  k   , )   entry 

in  the  win-loss  matrix  or  its  transpose   (k  ,,,k  )   or  both 

^       s+1   s 

are  positive) .   If  two  instructors  appear  in  different  sets 
then  they  are  not  directly  comparable  themselves,  and  there 
exists  no  intermediate  string  of  pairwise  directly  comparable 
instructors  connecting  them.   In  this  way  the  class  of  instructors 
is  partitioned  into  sets.   Within  each  set  there  is  information 
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that  has  a  direct  bearing  on  the  question  of  ordering  the 
instructors  in  that  set.   For  instructors  belonging  to  distinct 
sets,  there  is  no  information  for  comparing  them. 

Scaling  methods  are  available  to  order  linearly  all 
instructors  within  each  set.   This  does  not  mean  that  such 
orderings  within  sets  are  unique.   If  the  original  data  are 
highly  coherent  (i.e.  little  controversy  among  the  voters  in 
forming  direct  comparisons)  then  different  ordering  methods 
will  produce  essentially  the  same  results.   Different  methods 
may  not  produce  the  same  orderings  within  a  subset  if  there 
is  substantial  disagreement  in  the  direct  comparisons  provided 
by  the  voters.   The  degree  of  coherence  (or  disagreement)  within 
an  individual  subset  can  be  indicated  in  two  ways:   1)  Compute 
a  'stress'  function  that  measures  the  comparative  difficulty 
in  assigning  positions  to  instructors,  and   2)  Perform  the 
linear  ordering  in  more  than  one  way  and  compute  a  measure 
of  correlation  or  concordance  of  the  results.   The  remainder 
of  this  section  deals  with  the  description  of  two  methods  for 
ordering  the  competitors  in  a  win-loss  array  and  a  comparison 
of  how  they  perform  when  applied  to  our  ballot  data. 

Classical  Scaling 

If   A   is  a  win-loss  matrix  then  the  (i,j)th  entry  of 

T 
N  =  A  +  A    (A  transpose)  is  the  total  number  of  contests 

between   i   and   j.   If   A   is  divided  by   N   elementwise,  the 
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i,jth  element  of  the  resultant,  p. .,  is  the  empirical  prob- 
ability that   i   is  preferred  to   j   (according  to  all  those 
who  have  examined  both) .   This  probability  is  converted  to  an 
approximately  normal  random  variable,  6 . . ,  by  use  of  the 
Freeman-Tukey  transformation  [6]. 

The  classical  scaling  model  Pef.  [8]  assumes  that  the 
positions  of  the  teachers,  T, ,  T~  ,  .  .  .   are  interval  scale 
quantities  on  a  normal  scale.   Thus  each   6. .   is  viewed  as 

an  estimate  (or  realization)  of   T.  -T..   The  scale  values 

i    D 

T, ,  T~  ,  .  .  .   are  estimated  by  means  of  a  weighted  least  squares, 

i.e  . , 

min     Ww..(e..-T.  +T.)2  (5.1) 

T, Tk  i^4   ID   XJ     i     J 

where  the  weights 

w.  .  =  N.  .  +  .5  (5.2) 

iD     i] 

are  inverse  variances  of  the   0. ..   A  unique  solution  is 

ID         ^ 

available  for  each  subset  of  comparable  instructors.   The 
partial  derivatives  with  respect  to   T  ,  when  set  equal  to 
zero,  lead  to  the  system  of  equations 

T   T  w  .  =  J  w  .  (6  .  +  T.)  ,_  _ 

r  L.      rj        %      rj   rj     j  (5.3) 


which  can  be  solved  iteratively,  initializing  all   T.  =  0. 

Let   {T.}   be  the  solution  and  define 

l 
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<t>.  =   y   w.  .  (6.  .  -  T.  +  T  .)  (5.4) 

i    j£i   ID   ID     i     D 

as  the  stress  value  for   T.  ,  i.e.,  it  is  a  measure  of  the 
difficulty  in  fitting   T.   into  a  linear  scale  containing  the 
others.   Note  that  the  total  stress  \   4> .   is  the  originally 
minimized  objective  function.   If  it  is  near  zero,  then  the 
data  are  well  behaved.   If  it  is  large  then  the  data  are 
being  forced,  under  considerable  stress,  into  a  line.   The 
quantity 

HVlWijl172  (5-5) 

is  a  pooled  standard  deviation  of  the  residuals  of  the  scaling 

intervals   T.  -  T . . 
i    D 

Ford  Weights 

The  model  of  Lester  Ford,  Jr.,  Ref.  [5],  assumes  the 
existence  of  a  system  of  weights  or  "odds"  {w.}   having  the 
property  that,  p.  .,  the  probability  that   i   is  preferred  to   j, 
is  given  by 

p..=w./(w.  +w.)  (5.6) 

iD     ii     D 

Under  the  assumption  that  the  contests  represented  by  the 
data  in  the  win-loss  matrix  are  independent  contests,  the 
weights  {w.}  can  be  estimated  by  maximum  likelihood.  A 
unique  solution  will  exist  for  each  set  of  comparable  instructors 
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and  it  can  be  found  by  iterative  techniques.   The  log  of  the 
likelihood  function  is 

In  L=  H  a..[kw.  -  in(w.  +  w.)]  (5.7) 

where   (a  .}  are  the  elements  of  the  win-loss  matrix.   The 
iD 

system  of  partial  derivatives  of  (5.7)  with  respect  to   w 
leads  to  the  system  of  equations 


J"  a    =  w   y  N  ./(w   +  w.)  (5.8) 

h      rn     r  .   rn    r    i 
3         J       3         J 


which  can  be  solved  by  iteration  using  the  "win  percentages" 
of  the  instructors  to  initialize  the   (w  } .   The  quantities 


<j>.  =  -  Y  a.  .  [in   w   -  £n(w.  +  w  .)  ]  (5.9) 

i      j   13      i        !     3 


can  be  used  for  measuring  the  stress  in  estimating  w. .  If  the 
total  stress  is  large  then  the  likelihood  function  is  flat  near 
the  maximum. 

Both  the  Ford  weights  and  the  classical  scaling  models 
were  applied  to  the  19  79  data.   Due  to  space  limitations  in 
the  computer,  the  application  was  limited  to  a  60  by  60  win-loss 

matrix  consisting  of  the  top  60  ranked  instructors  of  1979 

* 

ranked  according  to  the  newly  proposed  score   S    (with   p  =  . 75 

and  weights  5:4:3:1).   This  class  of  instructors  formed  a 
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27 

9  0 

"0.03 

2.  19 

24 

78 

"0.  03 
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0.87 
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10 
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48 
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1  .57 

2? 
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3C 

72 

"0.09 

1  .88 

50 

32 

"0.  10 

0.93 

4  1 

39 

"0.  11 

1  .06 
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41 
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0.66 

3  1 

61 

"0.  12 

1  .96 

9 

3  6 

"0.  13 

1.44 

14 

9  4 

"0.  16 

1  .23 
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37 
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0.  39 

4C 

eo 

~0.  18 

2  .91 

4  2 

61 

"0.  20 

1  .43 

5" 

36 

"0.  20 

1  .46 

5£- 

37 

"0.21 

0.78 

GC 

112 

"0.21 

4.42 

43 

85 

"0.  22 

3.61 

4C 

09 

**0.  25 

2  .  18 

37 

26 

"0.51 

1  .85 

rd  Model 

CS  =  Score  by  Classical  Scaling 
ID  =  Instructor  identification:  rank  by  basic 
1979  scores  S* 
Stress  =  %  of  total  stress. 

TABLE  5.1.   Summary  of  Paired  Comparisons  Study 
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single  set  of  comparable  instructors.   The  results  appear 
in  Table  5.1.   The  two  methods  agree  quite  well  within  them- 
selves (Kendall's  coefficient  of  concordance  is  0.98).   The 
total  stress  is  rather  large  for  each  (the  standard  deviation 
in  (5.5)  is  0.48).   Each  of  the  scalings  in  Table  5.1  are  pre- 
sented in  top-to-bottom  order.   The  individual  instructors  are 
identified  by  their  ID  numbers  which  are  their  ranks  (top- to- 
bottom)  using  (4.3).   See  Appendix  C  for  further  summaries  of 
the  19  79  data. 

Although  the  two  scaling  methods  agree  within  themselves, 

they  are  in  only  modest  agreement  with  the  ranking  provided  by 

* 

the  score   S  .   It  was  decided  to  vary  the  inputs  (weights 

* 
w  and  exponent   p)   of     S    to  see  if  this  condition  could 

be  improved.   The  results  appear  in  Table  5.2  which  contains 

the  values  of  Kendall's  tau. 


Wts      p 

1.0 

.75 

.r 

WO  =  3:2:1:0 

.44/. 45 

.42/. 43 

. 32/. 33 

Wl  =  5:4:3:1 

.49/. 49 

.41/. 42 

.27/. 28 

W2  =  4:3:2:1 

.4  8/. 4  8 

.40/. 41 

.26/. 27 

W3  =  1:1:1:0 

.48/. 49 

.4 3/. 4  4 

.29/. 30 

W4  =  1:1:1:1 

.56/. 56 

.26/. 25 

.06/. 07 

Kendall's   t   Computed  Between   S    and  FW/CS 


TABLE  5.2.   Concordance  Coefficients 

19  79  Data 
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Examination  of  this  table  shows  that  the  best  agreement 

* 

of   S   with  paired  comparison  scalings  occurs  at  the  original 

normalization,  p  =  1.   This  in  turn  suggests  that  this  scaling 
also  has  a  bias  favoring  instructors  with  small  values  of   N. 
On  the  other  hand,  the  changes  in  instructor  position  provided 

by  the  paired  comparison  scoring  provides  information  that  is 

* 
unavailable  in   S  .   When  an  instructor  who  is  ranked  low  down 

* 
according  to   S    appears  high  on  a  paired  comparison  list,  we 

* 
know  that  his  low  value  of   S    is  due  to  the  fact  that  he  has 

been  compared  with  a  strong  set  of  instructors  on  the  ballots. 

Similarly  when  we  see  an  instructor  who  is  ranked  high  on  the 

* 

S    list  appear  low  down  on  a  paired  comparison  list  we  know 

that  he  faced  weaker  competition  on  the  ballots. 

* 
Since  it  is  difficult  to  get  the  advantages  of   S 

and  paired  comparisons  together  in  a  single  scoring  system, 
it  seems  wise  to  include  both  in  the  data  summaries  to  be  re- 
viewed by  the  selection  committee.   However,  the  Ford  weights 
and  the  classical  scaling  are  not  both  needed.   The  latter  is 
selected  arbitrarily  for  use  in  the  proposed  data  summary 
system. 
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VI.   CONCLUSIONS  AND  RECOMMENDATIONS 

Generally,  the  award  has  a  good  reputation.   The  system 

has  always  selected  a  good  teacher  for  the  recipient.   The 

practice  of  choosing  as  winner  an  instructor  with  a  good 

"past  performance"  record  in  the  balloting  is  a  good  one,  but 

changes  are  needed  to  remove  biases  (with   N)  when  considering 

past  performance.   The  ballot  validation  restriction  of 

5  <  K  <  25   is  arbitrary  but  wise  since  it  reduces  the  negative 

effect  of  a  voter  listing  a  large  number  of  instructors. 

Further  adjustment  for  this  effect  is  possible  (using  (4.2)  with 

w.  >  0)   and  is  considered  worthwhile. 
4 

In  order  for  the  system  to  maintain  its  dignity  the 
selection  committee  should  monitor  the  responses  in  terms  of 
watching  the  distributions  of  ballots  returned  by  curricular 
area  relative  to  the  number  sent  out,  the  distributions  of 
number  of  faculty  listed  on  ballots  by  curricula  area,  and 
distribution  of  ballots  to  alumni  by  curricula  area.   Inbalances 
in  these  distributions  should  be  avoided  and  investigated 
when  they  occur  as  they  may  signal  the  presence  of  unfair 
practices.   Computer  programs  are  being  written  to 
provide  data  summaries  of  these  distributions. 

Because  of  the  arbitrary  inputs,  the  ranking  of  instruc- 
tors is  arbitrary  and  it  is  nonsensical  to  make  firm  distinctions 
for  the  historical  record.   The  balloting  system  is  geared  to 
identify  good  instructors.   It  is  reasonable  to  select  a 
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"top  5%"  and  a  "next  15%"  for  storage--the  former  being 
marked  with  an  "A"  and  the  latter  with  a  "B" .   All  other 
eligible  faculty  should  be  identified  with  an  "E"  and  ineligible 
faculty  with  an  "I".   The  values  of   N   should  be  kept  also. 
A  set  of  IBM  cards  is  being  prepared  to  contain  this  record. 
Also  a  computer  program  is  being  written  to  facilitate  the 
yearly  updating  of  the  record. 

The  exponent   p   and  weights   W  =  (w  :w~:w_:w4)   of 
the  new  score   S    are  arbitrary  inputs.   The  values   p  =  .75 
and  W  =  (5:4:3:1)  were  chosen  by  the  19  78  Selection  Committee 
for  the  following  reasons: 

a)  The  exponent   p  =  . 75  serves  to  avoid  a  correlation 
with   N   of  the  high  scores. 

b)  The  weights   W  =  (5:4:3:1)   provide  equally  spaced  weights 
for  the  ranking  positions  and  also  allows  a  positive 
contribution  to  the  score  of  eligible  faculty  who  are 
listed  on  a  ballot  but  not  ranked.   The  policy  helps  to 
offset  the  manipulative  effect  of  ballots  that  list  a 
large  number   K   of  instructors.   See  Equation  (4.3). 

Future   Selection  Committees  should  monitor  the  performance 

of  the  inputs   p   and   W.   Computer  programs  that  produce  output 

similar  to  that  in  Appendix  A  are  being  written  for  this  purpose 

Finally,  the  data  summaries  supplied  to  the  Selection 
Committee  will  consist  of  three  pages  of  output: 
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(i)   For  each  of  the  top  60  instructors  (according  to  the 

* 
score   S  )  will  be  listed  (see  Appendix  D) 

* 
Rank  by  decreasing  values  of   S 

N:   the  number  of  ballots  that  identify  him 

D:   the  number  of  ballots  that  provide  a 

supporting  statement 

* 

S  :   the  score 

X, ,X2,X3:   the  total  accumulated  1st,  2nd  and  3rd 
place  votes 
Z,  , Z~ , Z_, Z.  :   the  total  accumulated  quantile  counts 
of  the  four  types  of  responses 
(ii)   For  each  string  of  comparable  instructors  in  the  top 
60  will  be  listed  (see  Table  (5.1)) 

Rank  according  to  localized  (and  decreasing)  CS 

* 
Rank  (ID)  according  to   S 

N:   the  number  of  ballots  that  identify  him 

CS :   score  by  classical  scaling 

Stress:   as  a  percentage  of  total  stress  for 

that  string. 

(iii)   For  each  of  the  top  30  instructors  will  be  listed 

(similar  to  Appendix  D) 

* 
Current  values  of  N,  S  ,  Rank  by  CS  and 

string  identification 

Recorded  values  of  N  and  grade  code 

(A,B,E,I)  for  each  of  the  past 

seven  years. 
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It  is  expected  that  the  first  page  will  provide  the 
committee  with  adequate  numerical  information  so  that  its 
members  can  visualize  the  entire  body  of  data.   This  can 
serve  as  a  general  basis  of  comparison.   The  second  page 
will  help  identify  such  instructors  who  have  a  relatively  low 
S*  because  they  found  strong  competition  or  who  have  a 
relatively  high   S*   because  they  faced  weaker  competition. 
The  past  performance  data  is  contained  on  page  3  together 
with  excerpts  from  the  first  two  pages.   The  information 
on  these  three  pages  provides  the  basis  for  designating  a 
winner,  the  top  5%  of  the  eligible  instructors,  and  the 
next  15%. 
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APPENDIX  A 

Further  study  of  the  distribution  of   N   (number  of 
ballots  that  identify  the  instructor)  as  it  relates  to  the 
large  scores   S   (S   =  (w.^  +  w2Z2  +  w3Z3  +  w4  Z^ )  *  np)  . 

To  set  the  standard,  the  overall  distribution  of   N 
for  1978  is  summarized  as  follows. 


MIN 

LOQ 

MEDIAN 

UPQ 

MAX 

SIZE 

1.00 

15.50 

28.00 

42.00 

101.00 

212.00 

where   LOQ   and   UPQ   stand  for  lower  and  upper  quartiles, 
respectively.   The  above  format  is  the  output  of  an  APL  program 
called  CONDENSE  and  is  used  liberally  in  what  follows.   See  [9]. 

More  specifically,  a  program  called  DECOMP  was  written 
which  takes  the  top  60  scores   S    (for  given  weights   W   and 
power   p) ,   decomposes  them  into  six  sets  containing  10  instructors 
each  (according  to  decreasing   S  ) ,  and  summarizes  the  corre- 
sponding six  sets  of  10  values  of   N.   These  summaries  are 
in  the  form  of  CONDENSE  and  each  is  followed  by  a  set  of  vertical 
BOXPLOTS  (Ref.  [9]).   The  following  seven  figures  contain  the 
results.   The  choice   p  =  .75  is  supported  rather  generally  as 
a  scoring  scheme  whose  high  scores  favor  neither  large  nor  small 
values  of  N.   It  also  appears  that  attention  should  be  drawn  to 
Fig.  A-7,  especially  in  the  light  of  the  behavior  of  the  weight 
system  W3  =  (1:1:1:0)   coupled  with   p  =  1   in  Table  5.2. 
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10.  00 

23.  00 
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52.  00 
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1*1 

I  I 
I  I 

~T~ 
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X 

_1_ 

I   I 
i*i 

T 
x 

X 


_1_ 
1  =  1 


FIGURE    A-l 


S*    =     (5Z1    +    4Z2    +    3Z3    +    Z4)     f    N*5 
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FIGURE    A-2 


*  1*> 

S      =    (5Zn    +    4Z^    +    3Z      +    Z    )     t    N* 

12  3  4 
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FIGURE   A-3 


S*   =    (4ZX    +    3Z2    +    2Z3    +    Z4  )     t    N*5 
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FIGURE   A-4 


S      =     (4ZX    +    3Z2    +    2Z       +    Z    )     v    N*75 
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FIGURE    A- 5 


S      =    (4Z       +    3Z2    +    2Z3    +    Z    )     t    N 


39 


10  DECO, 
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FIGURE    A-6 


S*    =     (Z1    +    Z2     +    Z    )     t    N*5 
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FIGURE   A- 7 


S      =     (Z1    +    Z2    +    Z3)     t    N 
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APPENDIX  B 
EFFECT  OF  VARYING  INPUTS 

The  1978  data  were  used  to  explore  the  effect  of 

varying  the  exponent   p   and  the  weighting  system  W. 

Specifically, 

p  =  .  75 ,  .5 

and 

WO  =  3:2:1:0 

Wl  =  5:4:3:1 

W2  =  4:3:2:1 

W3  =  1:1:1:0 

W4  =  1:1:1:1 


The  index  numbers  in  Table  B-l  are  the  ranks  according  to  the 
original  score   S   with   p  =  1   and   W  =  4:2:1.   Table  B-2 
contains  the  corresponding  values  of   N. 
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p  -  .75 


p.-  .5 


Rank 

1 
2 
3 
4 

5 

6 

7 

8 

9 
10 
11 
12 
13 
14 
15 
16 
17 
10 
19 
20 
21 
22 
23 
24 
25 
26 
27 
28 
29 
30 
31 
32 
33 
3  4 
35 
36 
37 
38 
39 
40 
41 
42 
43 
44 
45 
46 
47 
48 
49 
50 
51 
52 
53 
54 
55 
56 
57 
58 
59 
CO 


WO 

Wl 

W2 

W3 

W4 

4 

4 

4 

4 

4 

2 

2 

2 

5 

10 

1 

5 

5 

10 

7 

5 

10 

10 

2 

5 

3 

1 

1 

1 

18 

10 

7 

7 

7 

21 

7 

3 

3 

3 

28 

11 

11 

11 

11 

11 

in 

18 

18 

8 

32 

15 

15 

15 

13 

15 

8 

21 

21 

18 

2 

1? 

8 

0 

15 

36 

16 

16 

12 

21 

24 

21 

12 

28 

16 

31 

9 

28 

16 

19 

33 

28 

13 

32 

32 

8 

13 

32 

9 

33 

16 

6 

9 

24 

9 

3 

14 

24 

13 

28 

12 

24 

19 

14 

12 

CO 

19 

14 

19 

30 

37 

32 

33 

31 

14 

23 

31 

31 

33 

24 

48 

33 

30 

36 

27 

13 

23 

23 

23 

6 

57 

27 

6 

6 

34 

19 

36 

27 

27 

2  3 

55 

20 

30 

37 

31 

3  4 

30 

37 

30 

37 

30 

37 

34 

34 

26 

1 

26 

48 

4  8 

3G 

27 

25 

GO 

GO 

55 

14 

3<4 

26 

20 

60 

70 

22 

20 

2G 

40 

9 

40 

55 

2  5 

48 

61 

35 

40 

72 

70 

76 

40 

25 

35 

20 

53 

39 

35 

39 

4  2 

71 

60 

22 

40 

35 

85 

42 

42 

57 

61 

45 

53 

39 

53 

25 

4  4 

29 

53 

55 

39 

6  7 

44 

57 

42 

22 

39 

45 

44 

44 

G7 

40 

57 

Gl 

45 

53 

91 

55 

45 

61 

52 

77 

52 

52 

5G 

44 

70 

56 

5G 

52 

77 

42 

61 

71 

71 

76 

107 

54 

70 

29 

79 

5C 

71 

76 

76 

57 

127 

41 

67 

79 

41 

35 

58 

79 

G7 

85 

52 

67 

29 

54 

71 

104 

38 

85 

70 

5G 

126 

76 

54 

85 

GO 

118 

70 

77 

58 

20 

54 

47 

58 

77 

4  5 

73 

17 

G3 

G3 

0  4 

25 

51 

73 

51 

81 

26 

WO 

Wl 

W2 

W3 

W4 

4 

4 

4 

4 

20 

10 

10 

10 

10 

4 

5 

7 

18 

5 

10 

7 

5 

7 

7 

10 

11 

10 

5 

18 

32 

18 

11 

11 

11 

21 

28 

28 

20 

21 

7 

2 

21 

21 

20 

11 

15 

15 

15 

32 

36 

21 

3? 

'   32 

15 

60 

3 

24 

24 

8 

24 

32 

2 

36 

2 

31 

24 

36 

2 

33 

15 

12 

3 

31 

16 

5 

16 

31 

3 

3 

48 

8 

16 

16 

24 

33 

1 

12 

12 

13 

57 

36 

8 

8 

19 

55 

31 

33 

33 

36 

37 

27 

60 

GO 

60 

71 

33 

27 

27 

31 

76 

9 

48 

48 

27 

61 

19 

19 

23 

12 

16 

23 

23 

37 

55 

27 

14 

37 

19 

30 

79 

13 

1 

1 

1 

126 

48 

9 

0 

37 

23 

60 

13 

57 

0 

12 

37 

55 

55 

2  3 

85 

6 

14 

14 

48 

127 

26 

57 

13 

34 

135 

34 

34 

34 

14 

53 

25 

30 

53 

26 

8 

30 

61 

61 

61  • 

91 

57 

26 

71 

4  0 

141 

55 

53 

3  0 

70 

107 

20 

71 

45 

7  6 

123 

35 

76 

2  6 

57 

115 

53 

44 

4  4 

71 

34 

22 

4  5 

76 

35 

45 

45 

25 

25 

53 

118 

4  4 

35 

35 

79 

44 

61 

79 

79 

44 

159 

39 

56 

5  6 

6 

129 

40 

40 

20 

85 

19 

71 

20 

30 

67 

77 

56 

30 

22 

4  2 

G7 

42 

85 

85 

77 

56 

52 

6 

40 

20 

104 

7G 

22 

6 

25 

82 

54 

52 

52 

52 

101 

70 

67 

67 

5  6 

145 

29 

42 

42 

3  0 

3 

67 

70 

54 

22 

30 

58 

77 

70 

45 

70 

63 

54 

77 

104 

6  3 

es 

01 

6  3 

69 

2 

70 

6  3 

01 

82 

•  54 

77 

e2 

02 

0  4 

no 

51 

73 

73 

01 

73 

TABLE  B-l.   Index  Numbers  for  Several  Scoring  Systems 
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• 

p  -  .75 

p  -  .5 

Rank 

wo 

Wl 

W2 

H3 

W4 

WO 

Wl 

W2 

W3 

W4 

1 

58 

58 

58 

58 

50 

58 

58 

58 

50 

101 

2 

13 

13 

13 

36 

71 

71 

71 

71 

71 

58 

3 

6 

36 

36 

71 

48 

36 

4fl 

6fl 

36 

71 

4 

36 

71 

71 

13 

36 

48 

36 

48 

48 

68 

5 

18 

6 

6 

6 

68 

57 

e? 

36 

6  8 

7  8 

6 

71 

40 

48 

48 

63 

6fi 

57 

57 

57 

63 

7 

48 

18 

18 

10 

101 

101 

101 

101 

6  3 

48 

8 

57 

57 

57 

57 

57 

13 

63 

63 

101 

57 

9 

68 

68 

68 

26 

78 

48 

4R 

48 

78 

76 

10 

48 

48 

4  8 

18 

48 

63 

78 

78 

48 

83 

11 

26 

63 

63 

68 

13 

18 

6  6 

66 

26 

66 

12 

32 

26 

26 

4G 

76 

70 

13 

76 

13 

63 

13 

35 

35 

32 

63 

66 

66 

7  6 

13 

"5 

48 

14 

63 

32 

101 

35 

63 

32 

IB 

63 

35 

36 

15 

20 

101 

35 

28 

45 

35 

6  3 

19 

18 

6  6 

16 

101 

18 

78 

78 

26 

26 

35 

35 

6  6 

45 

17 

18 

78 

20 

45 

35 

6 

32 

32 

18 

6  1 

18 

11 

20 

66 

20 

18 

76 

26 

26 

28 

59 

19 

20 

60 

18 

101 

32 

63 

45 

45 

76 

46 

20 

66 

28 

20 

32 

83 

50 

83 

83 

83 

61 

21 

28 

20 

28 

25 

46 

45 

50 

50 

6  3 

59 

22 

78 

45 

63 

20 

38 

20 

66 

66 

50 

53 

23 

G3 

6  3 

45 

66 

66 

28 

28 

38 

32 

35 

24 

45 

76 

76 

50 

18 

38 

38 

46 

59 

50 

25 

38 

38 

38 

11 

61 

20 

46 

28 

25 

50 

26 

50 

11 

11 

33 

28 

18 

6 

6 

6 

75 

27 

76 

50 

50 

38 

59 

66 

20 

20 

46 

38 

28 

22 

25 

46 

63 

33 

83 

18 

61 

20 

32 

29 

25 

46 

25 

4  6 

25 

46 

59 

59 

38 

49 

30 

46 

33 

33 

32 

6 

11 

20 

20 

6  6 

64 

31 

32 

66 

66 

76 

50 

32 

61 

10 

33 

74 

32 

30 

83 

03 

59 

20 

33 

33 

33 

20 

45 

33 

33 

32 

22 

03 

50 

30 

25 

45 

32 

26 

34 

24 

22 

32 

25 

20 

25 

53 

53 

53 

52 

35 

66 

59 

30 

66 

53 

61 

32 

61 

25 

73 

36 

30 

25 

24 

34 

59 

59 

45 

25 

3  4 

52 

37 

25 

30 

30 

22 

45 

22 

61 

4  3 

59 

70 

38 

27 

30 

27 

23 

61 

30 

59 

32 

61 

65 

39 

83 

24 

25 

3  0 

49 

45 

42 

42 

61 

3  3 

40 

23 

23 

61 

53 

43 

24 

43 

59 

30 

43 

41 

45 

27 

45 

30 

4  2 

43 

30 

30 

45 

56 

42 

16 

45 

59 

27 

37 

42 

30 

30 

50 

4? 

43 

42 

CI 

23 

24 

27 

53 

50 

50 

42 

59 

44 

43 

42 

42 

37 

25 

27 

42 

42 

11 

6  5 

45 

61 

53 

43 

45 

52 

25 

25 

^2 

49 

28 

4  0 

59 

43 

53 

33 

4  1 

6  1 

22 

27 

3  7 

41 

47 

33 

"33 

42 

4? 

34 

42 

27 

2  4 

23 

37 

48 

42 

42 

33 

41 

23 

23 

49 

4  9 

41 

4? 

49 

53 

61 

61 

59 

52 

3  3 

11 

25 

22 

4  6 

50 

38 

3  4 

16 

50 

4? 

5^ 

2  4 

11 

30 

4  7 

51 

61 

50 

59 

61 

G4 

38 

33 

33 

3  3 

51 

52 

7 

37 

50 

7 

30 

50 

37 

37 

42 

50 

53 

31 

50 

37 

49 

3  3 

16 

23 

23 

27 

18 

54 

37 

16 

38 

61 

46 

37 

34 

38 

24 

25 

55 

11 

49 

34 

42 

75 

31 

4  1 

3  4 

43 

34 

5C 

59 

3G 

49 

25 

56. 

41 

30 

41 

46 

41 

57 

34 

41 

31 

16 

39 

49 

52 

41 

25 

13 

58 

17 

31 

41 

43 

37 

34 

41 

52 

47 

38 

59 

3 

41 

41 

30 

30 

41 

i*7 

47 

30 

41 

CO 

27 

37 

27 

2 

32 

27 

37 

37 

52 

37 

TABLE  B-2.   Values  of   N 


Corresponding  to  Table  B-l . 
44 


APPENDIX  C 

DATA  SUMMARY  OF  TOP  SIXTY  SCORES  BY  CURRENT  METHOD 
AND  BY  THE  PROPOSED  NEW  METHOD 


45 


Rank      S       D     N     XI      X2     X3 


1 

3.33 

3 

c 

4 

2 

0 

2 

2.69 

6 

13 

8 

1 

1 

3 

2.22 

C 

10 

8 

4 

0 

u 

2.00 

17 

58 

24 

9 

2 

5 

1.92 

9 

36 

12 

a 

5 

6 

1.73 

4 

11 

4 

l 

1 

7 

1.  50 

g 

4T 

1^ 

7 

6 

8 

1.  58 

5 

26 

7 

4 

5 

3 

1.  55 

4 

20 

5 

5 

1 

10 

1.51 

12 

71 

18 

15 

7 

11 

1.51 

9 

57 

1G 

9 

4 

12 

1.50 

6 

32 

11 

1 

2 

13 

}.  50 

2 

18 

3 

5 

5 

14 

1.40 

4 

20 

5 

3 

2 

15 

1.38 

10 

48 

13 

4 

6 

16 

1.37 

4 

35 

7 

9 

2 

17 

1.33 

1 

3 

1 

0 

0 

18 

1.32 

4 

68 

16 

11 

4 

19 

1.25 

5 

28 

5 

5 

5 

20 

1.14 

4 

22 

5 

2 

1 

21 

1.13 

6 

63 

10 

12 

7 

22 

1.08 

4 

24 

5 

3 

0 

23 

1.05 

5 

38 

7 

5 

2 

24 

1.05 

6 

66 

13 

7 

3 

25 

1.03 

5 

30 

7 

1 

1 

26 

1.03 

2 

32 

5 

4 

5 

27 

1.02 

6 

50 

8 

7 

5* 

28 

1.01 

13 

101 

18 

10 

10 

29  . 

1.  00 

2 

16 

3 

2 

0 

30 

1.00 

2 

25 

2 

6 

5 

31 

0.97 

7 

63 

11 

7 

3 

32 

0.96 

4 

78 

9 

16 

7 

33 

0.96 

4 

45 

4 

11 

5 

34 

0.94 

3 

33 

4 

5 

5 

35 

0.93 

3 

30 

5 

3- 

2 

36 

0.92 

10 

76 

15 

2 

6 

37 

0.91 

5 

46 

6 

6 

6 

38 

0.91 

2  • 

11 

2 

1 

0 

39 

0.89 

4 

27 

5 

1 

2 

40 

0.88 

0 

25 

3 

3 

4 

41 

0.06 

0 

7 

0 

3 

0 

42 

0.83 

1 

23 

2 

5 

1 

43 

0.80 

1 

5 

1 

0 

0 

44 

0.  79 

3 

42 

6 

3 

3 

45 

0.  77 

4 

43 

7 

2 

1 

46 

0.  76 

1 

17 

3 

0 

1 

47 

0  .  76 

2 

17 

3 

0 

1 

43 

0  .  76 

5 

6G 

6 

12 

2 

4  9 

0  .  75 

1 

8 

1 

1 

0 

50 

0  .75 

3 

16 

3 

0 

0 

51 

0.  74 

3 

27 

5 

0 

0 

52 

0.73 

2 

33 

3 

5 

2 

53 

0.71 

4 

45 

4 

8 

0 

54 

0.  71 

5 

38 

6 

1 

1 

55 

0.69 

4 

50 

6 

.   2 

13 

50 

0.69 

3 

42 

5 

3 

3 

57 

0.69 

3 

61 

8 

3 

4 

58 

0.C8 

2 

31 

4 

2 

1 

59 

0.67 

4 

24 

4 

0 

0 

60 

0.66 

4 

83 

6 

u 

9 

TABLE  c-1.   Score  Summary  1978.   (Old  System  W  =  4,2,1;  p = 1) . 
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Rank 


Zl 


Z2 


Z3 


Z4 


XI 


X2 


X3 


1 

58 

17 

7.10 

2  1.61 

7.20 

1.63 

fl.90 

2  4 

9 

2 

2 

13 

c 

0.56 

7.00 

0.90 

0.70 

1.21 

0 

1 

1 

3 

36 

9 

0.4  9 

10.16 

0.70 

3.  32 

3.78 

12 

8 

5 

4 

71 

12 

G.  34 

10  .  30 

11.34 

5.  32 

9.12 

18 

15 

7 

5 

C 

3 

c.rs 

3.03 

1.49 

0.00 

0.00 

4 

2 

0 

C 

4  6 

9 

0.06 

12.89 

5.  76 

5.12 

7.6  0 

14 

7 

6 

7 

18 

6 

5. PS 

7.11 

3.42 

0.00 

2.14 

0 

4 

0 

8 

57 

9 

s.oo 

1 4  .  n  2 

7.49 

2.2  4 

9.71 

16 

3 

4 

9 

68 

4 

5.  52 

14.47 

9.  13 

2.85 

13.  27 

16 

11 

4 

10 

4  8 

10 

5.3  2 

12.  02 

3.  58 

4.6  2 

8.65 

13 

4 

6 

11 

63 

6 

5.14 

6.13 

10.  33 

5.  39 

12.71 

10 

12 

7 

12 

26 

5 

5.13 

6.  32 

3.  07 

3.  86 

3.  59 

7 

4 

5 

13 

35 

4 

4  .88 

6.23 

7.21 

1.  38 

6  .10 

7 

9 

2 

14 

32 

6 

4.87 

10.16 

0.82 

1.65 

6.  54 

11 

1 

2 

15 

101 

13 

4.83 

16.16 

7.69 

6.76 

22.08 

18 

10 

10 

1G 

18 

2 

4.71  * 

2.58 

3.98 

3.49 

1.94 

3 

5 

5 

17 

7  8 

4 

4.67 

8.08 

12.46 

5.  55 

15.71 

9 

16 

7 

18 

20 

4 

4.59 

4.  30 

4  .10 

0.63 

3.34 

5 

5 

1 

19 

66 

6 

4.49 

11.68 

6  .13 

2.20 

14.48 

13 

7 

3 

20 

28 

5 

4.46 

4  .  52 

4  .07 

3.77 

4.05 

5 

5 

5 

21 

20 

4 

4.41 

4  .60 

2.  51 

1.67 

3.71 

5 

3 

2 

22 

45 

4 

4.38 

3.67 

9.15 

4.17 

8.67 

4 

11 

5 

23 

63 

7 

4.23 

9.97 

5.  80 

2.08 

15.25 

11 

7 

3 

24 

7  6 

10 

4.15 

13.62 

1.64 

u  .47 

18. «2 

15 

2 

6 

25 

38 

5 

4  .12 

6.27 

4  .  51 

1.63 

8.72 

7 

5 

2 

26 

11 

4 

4  .10 

3.67 

0.03 

0.  50 

1.59 

4 

1 

1 

27 

50 

6 

4.05 

7.16 

5.  38 

3.29 

9.02 

8 

7 

5 

28 

25 

2 

3.97 

1  .77 

4  .99 

3.69 

u  .47 

2 

6 

5 

29 

4  6 

5 

3.86 

5.  34 

4.73 

4.01 

10.  55 

6 

6 

6 

30 

33 

3 

3. El 

3.  55 

4  .  31 

3.43 

7.10 

4 

5 

5 

31 

6  6 

5 

3.66 

5.42 

9.  58 

1.20 

15.67 

6 

12 

2 

32 

83 

4 

3.62 

5.  51 

8.  55 

6  .  51 

18.39 

6 

11 

9 

33 

32 

2 

3.01 

4  .  34 

3.24 

2.96 

5.10 

5 

4 

5 

34 

22 

4 

3.55 

4  .  54 

1  .  60 

0  .73 

4  .80 

5 

2 

1 

35 

5  9 

4 

3.45 

5.  56 

1  .42 

9.23 

12.  33 

6 

2 

13 

36 

2  5 

0 

3.45 

2  .81 

2  .47 

3.05 

5.43 

3 

3 

4 

37 

30 

5 

3.44 

6.39 

0.67 

0.83 

■  6.91 

7 

1 

1 

38 

30 

3 

3.41 

4.60 

2.3  3 

1.  56 

6  .60 

5 

3 

2 

39 

2«* 

4 

3.  36 

4.55 

2.05 

1.00 

5.  52 

5 

3 

0 

40 

2  3 

1 

3  .  33 

1.  89 

4  .  33 

0  .  82 

5.78 

2 

5 

1 

41 

2  7 

4 

3.  32 

4.64 

0.8  0 

1  .  74 

7.4  3 

5 

1 

2 

4? 

45 

4 

3.29 

3.75 

0  .  59 

1.  00 

12.15 

4 

8 

0 

43 

61 

3 

3.28 

7.  3P 

2.25 

2.8  6 

17.  14 

8 

3 

4 

4  4 

42 

3 

3.23 

5.  27 

2.53 

2.10 

1.0.7" 

6 

3 

3 

45 

53 

1 

3.21 

2  .70 

6.50 

3.33 

13.15 

3 

3 

5 

4  0 

4  3 

H 

3  .16 

r. .  ii  3 

1.  5.3 

0.  79 

12.18 

7 

2 

1 

47 

33 

2 

.3  .07 

2  .70 

4.26 

l  .23 

P  .  07 

3 

5 

2 

40 

42 

3 

3  .  04 

-..[3 

2.42 

2  .  12 

10.64 

5 

3 

3 

49 

61 

2 

3.01 

3.  59 

5.8  9 

2  .82 

15.79 

4 

7 

4 

5  0 

34 

0 

3.01 

0.  no 
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PREVIOUS  PERFORMANCE  DATA 
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A  =  top  5%,  B  =  next  15%,  E  =  Eligible,  I  =  Ineligible. 
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